fix(operator): wait for inclusion on rollup-message injection - #39
Open
devon-n wants to merge 2 commits into
Open
fix(operator): wait for inclusion on rollup-message injection#39devon-n wants to merge 2 commits into
devon-n wants to merge 2 commits into
Conversation
Both inject_direct_message call sites used `-w none` (fire-and-forget): the op was injected into the mempool but the call returned before it was baked, so it released advance_lock while the on-chain counter was still stale. octez-client computes the next op's counter from on-chain state, so the following injection from the operator key reused the same counter and L1 rejected it: cannot be added because the mempool already contains a conflicting operation That marked the shield `failed` (not retried); the next shield self-recovered. Switch both sites to `-w 1` so each op is baked before advance_lock releases — the counter always advances before the next injection is built, so the conflict can't occur. Costs ~1 block of latency per injection held under the lock. Test: attested_dal_submission_sends_pointer_message now asserts the pointer inject waits (`-w 1`, not `none`). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
devon-n
force-pushed
the
fix/operator-inject-wait-inclusion
branch
from
August 6, 2026 10:46
cc8b9b6 to
b9ce2df
Compare
Follow-up to waiting on inclusion (-w 1). Two hardening changes: Timeout: `-w 1` blocks until the op is baked, under advance_lock. Without a bound, one op that never confirms would hang the injection forever and wedge the whole operator (reconciler + every submit stall on the lock). Add a 180s ceiling in run_command_collect_output: capture stdout/stderr to temp files (pipes could deadlock on a long wait, and files let us keep partial output), poll try_wait to a deadline, kill on expiry, and return a timeout error that still carries whatever octez-client printed. Idempotency: octez-client prints the op hash the moment it injects, before it waits. So an injection can reach the node and *then* error (a confirmation timeout or RPC blip). Both inject_direct_message call sites now check for an op hash in the error: if present, the op is on the node, so record it as SubmittedToL1 instead of Failed / instead of propagating the error. This stops the reconciler from injecting a duplicate pointer op on the next tick. Inclusion is not verified here — a genuinely dropped op is still caught by the caller's settlement timeout. Tests: pointer_inject_error_with_op_hash_is_recorded_not_retried (idempotency), run_command_times_out_and_keeps_partial_output (timeout + partial-output). 33/33. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Shields intermittently fail with the operator recording:
Both
inject_direct_messagecall sites (process_submissiondirect inbox, and the DalPointer submit after attestation) usedoctez-client -w none— fire-and-forget. The op enters the mempool but the call returns before it is baked, releasingadvance_lockwhile the on-chain counter is still stale.octez-clientderives each op's counter from on-chain state, so the next injection from the operator key reuses the same counter and L1 rejects it. The shield is markedfailed(not retried); the next one self-recovers. ~3% of injections.Fix
1. Wait for inclusion. Both sites now use
-w 1, so each op is baked (counter advances) beforeadvance_lockreleases — the conflict can't occur.2. Bounded wait.
-w 1blocks under the lock, sorun_command_collect_outputgets a 180s ceiling (capture to temp files, polltry_wait, kill on expiry). Without it, one never-confirmed op would hang the operator indefinitely.3. Idempotent on post-injection error. octez-client prints the op hash the moment it injects, before waiting. If the wait then errors (timeout / RPC blip) the op already reached the node, so both call sites record it as
SubmittedToL1rather thanFailed/ re-injecting — otherwise the reconciler would inject a duplicate pointer op next tick.Cost / assumptions
advance_lock. Acceptable for the operator's throughput; removes the race deterministically.Test
cargo test -p tzel-services --bin tzel-operator→ 33/33; build + clippy clean. New:pointer_inject_error_with_op_hash_is_recorded_not_retried,run_command_times_out_and_keeps_partial_output, and a-w 1assertion onattested_dal_submission_sends_pointer_message.